Normal-gamma distribution

Normal-gamma
Parameters \mu\, location (real)
\lambda > 0\, (real)
\alpha \ge 1\, (real)
\beta \ge 0\, (real)
Support x \in (-\infty, \infty)\,\!, \; \tau \in (0,\infty)
PDF f(x,\tau|\mu,\lambda,\alpha,\beta) = \frac{\beta^\alpha \sqrt{\lambda}}{\Gamma(\alpha)\sqrt{2\pi}} \, \tau^{\alpha-\frac{1}{2}}\,e^{-\beta\tau}\,e^{ -\frac{ \lambda \tau (x- \mu)^2}{2}}
Mean [1] \operatorname{E}(X)=\mu\,\! ,\quad \operatorname{E}(\Tau)= \alpha \beta^{-1}
Variance [1] \operatorname{var}(X)= \frac{\beta}{\lambda (\alpha-1)} ,\quad \operatorname{var}(\Tau)=\alpha \beta^{-2}

In probability theory and statistics, the normal-gamma distribution is a bivariate four-parameter family of continuous probability distributions. It is the conjugate prior of a normal distribution with unknown mean and precision.[2]

Contents

Definition

Suppose

x|\tau, \mu, \lambda \sim N(\mu,1 /(\lambda \tau)) \,\!

has a normal distribution with mean \mu and variance 1 / (\lambda \tau) , where

\tau|\alpha, \beta \sim \mathrm{Gamma}(\alpha,\beta) \!

has a gamma distribution. Then (x,\tau) has a normal-gamma distribution, denoted as

(x,\tau) \sim \mathrm{NormalGamma}(\mu,\lambda,\alpha,\beta) \! .

Characterization

Probability density function

f(x,\tau|\mu,\lambda,\alpha,\beta) = \frac{\beta^\alpha \sqrt{\lambda}}{\Gamma(\alpha)\sqrt{2\pi}} \, \tau^{\alpha-\frac{1}{2}}\,e^{-\beta\tau}\,e^{ -\frac{ \lambda \tau (x- \mu)^2}{2}}

Properties

Scaling

For any t > 0, tX is distributed {\rm NormalGamma}(t\mu, \lambda, \alpha, t^2\beta)

Marginal distributions

By construction, the marginal distribution over \tau is a gamma distribution, and the conditional distribution over x given \tau is a Gaussian distribution. The marginal distribution over x is a three-parameter Student's t-distribution.

Posterior distribution of the parameters

Form of the posterior for a Normal random variable with a Normal-Gamma prior:

Presume the following hierarchy for a normal random variable X with unknown mean \mu and precision \lambda.

\begin{align} X & \sim \mathcal{N}(\mu, \lambda^{-1}) \\ \mu | \lambda &\sim \mathcal{N}(\mu_0, {(n_0 \lambda})^{-1}) \\ \lambda &\sim \mathcal{G}\left(\frac{\nu_0}{2},\frac{2}{S_0}\right) \end{align}

Where:

\mu_0 is the prior mean
S_0 is the prior sum of squared errors
n_0 is the prior sample size
\nu_0 is the prior degrees of freedom

Note the joint distribution of the parameters is Normal-Gamma. The posterior distribution of the parameters can be analytically determined by Bayes' rule working with the likelihood \mathbf{L(\lambda, \mu | X)}, and the prior \pi(\lambda, \mu ).

\begin{align} \mathbf{L(\lambda, \mu | X)} & \propto \prod_{i=1}^n \lambda^{1/2} \exp[\frac{-\lambda}{2}(x_i-\mu)^2] \\ & \propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\sum_{i=1}^n(x_i-\mu)^2] \\ & \propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\sum_{i=1}^n(x_i-\bar{x} %2B\bar{x} -\mu)^2] \\ & \propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\sum_{i=1}^n\left((x_i-\bar{x})^2 %2B (\bar{x} -\mu)^2\right)] \\ & \propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\left(S %2B n(\bar{x} -\mu)^2\right)] \end{align}

where S=\sum_{i=1}^n(x_i-\bar{x})^2, the sum of squared errors.

Now consider the prior,

\mathbf{\pi}(\mu,\lambda) \propto \lambda^{1/2}\exp[\frac{-\lambda n_0}{2}(\mu-\mu_0)^2] \lambda^{\frac{\nu_0}{2}-1}\exp[\frac{-\lambda S_0}{2}]

The posterior distribution of the parameters is proportional to the prior times the likelihood.

\begin{align} \mathbf{P(\lambda, \mu | X}) &\propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\left(S %2B n(\bar{x} -\mu)^2\right)] \lambda^{1/2}\exp[\frac{-\lambda n_0}{2}(\mu-\mu_0)^2] \lambda^{\frac{\nu_0}{2}-1}\exp[\frac{-\lambda S_0}{2}] \\ &\propto \lambda^{\frac{\nu_0%2Bn}{2}-1} \exp[\frac{-\lambda}{2}(S %2B S_0) ] \lambda^{1/2}\exp[\frac{-\lambda}{2}\left(n_0(\mu-\mu_0)^2 %2B n(\bar{x} -\mu)^2\right)] \\ \end{align}

Notice the right half begins to look like the kernel of a normal pdf and the left like a gamma. After a bit of juggling and completing the square the result will appear.

\begin{align} \mathbf{P(\lambda, \mu | X} )& \propto \lambda^{\frac{\nu_0%2Bn}{2}-1} \exp[\frac{-\lambda}{2}(S %2B S_0) ] \lambda^{1/2}\exp[\frac{- \lambda}{2} \left(n_0 (\mu^2 - 2 \mu \mu_0 %2B \mu_0^2 ) %2B n(\bar{x}^2-2 \mu \bar{x} %2B \mu^2)\right)] \\ & \propto \lambda^{\frac{\nu_0%2Bn}{2}-1} \exp[\frac{-\lambda}{2}(S %2B S_0 %2B n_0 \mu_0^2 %2B n \bar{x}^2) ] \lambda^{1/2}\exp[\frac{-\lambda}{2} (n%2Bn_0) \left(\frac{n_0 \mu^2 %2B n \mu^2 }{n %2B n_0} - 2 \mu \frac{n\bar{x} %2Bn_0\mu_0}{n%2Bn_0} \right)] \\ & \propto \lambda^{\frac{\nu_0%2Bn}{2}-1} \exp[\frac{-\lambda}{2}(S %2B S_0 %2B n_0 \mu_0^2 %2B n \bar{x}^2) ] \lambda^{1/2}\exp[\frac{-\lambda}{2} (n%2Bn_0) \left(\mu^2 - 2 \mu \frac{n\bar{x} %2Bn_0\mu_0}{n%2Bn_0} %2B \left (\frac{n\bar{x} %2Bn_0\mu_0}{n%2Bn_0}\right )^2 - \left (\frac{n\bar{x} %2Bn_0\mu_0}{n%2Bn_0}\right )^2\right)] \\ & \propto \lambda^{\frac{\nu_0%2Bn}{2}-1} \exp[\frac{-\lambda}{2}\left(S %2B S_0 %2B n_0 \mu_0^2 %2B n \bar{x}^2 - \frac{\left (n\bar{x} %2Bn_0\mu_0 \right )^2}{n%2Bn_0}\right) ] \lambda^{1/2}\exp[\frac{-\lambda}{2} (n%2Bn_0) \left ( \mu - \frac{n\bar{x} %2Bn_0\mu_0}{n%2Bn_0}\right )^2] \\ & \propto \lambda^{\frac{\nu_0%2Bn}{2}-1} \exp[\frac{-\lambda}{2}\left(S %2B S_0 %2B \frac{nn_0 (\bar{x}-\mu_0)^2}{n%2Bn_0}\right) ] \lambda^{1/2}\exp[\frac{-\lambda}{2} (n%2Bn_0) \left ( \mu - \frac{n\bar{x} %2Bn_0\mu_0}{n%2Bn_0}\right )^2] .\\ \end{align}

This is a normal gamma pdf with parameters \mathcal{NG} \left(\frac{n\bar{x} %2Bn_0\mu_0}{n%2Bn_0}, n%2Bn_0, \frac{\nu_0%2Bn}{2}, 2\left(S %2B S_0 %2B \frac{nn_0 (\bar{x}-\mu_0)^2}{n%2Bn_0}\right)^{-1} \right) .

The reference prior is the limiting case as

n_0, S_0, \mu_0 \rightharpoonup 0

and \nu_0 \rightharpoonup -1

Generating normal-gamma random variates

Generation of random variates is straightforward:

  1. Sample \tau from a gamma distribution with parameters \alpha and \beta
  2. Sample x from a normal distribution with mean \mu and variance 1/(\lambda \tau)

Related distributions

Notes

  1. ^ a b Bernardo & Smith (1993, p.434)
  2. ^ Bernardo & Smith (1993, pages 136, 268, 434)

References